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ABSTRACT 

This paper discusses the process of analysing online discussion and argues for the merits of mixed methods. Much 
research of online participation and e-learning has been either message focused or person focused analysis. The former 
covers methodologies such as content and discourse analysis, the latter interviewing and surveys. The paper discusses the 
strength and weaknesses of these approaches in the context of a study of an online social educational network for gifted 
students. Here interviews, questionnaire survey and content analysis were all used in order to explore the process of 
online discussion and the experience of taking part. The paper argues for a mixed approach in which different types of 
data can be compared and contrasted. Such triangulation is time-consuming but it allows for a comprehensive picture of 
the use of the network and the experience of online participation. 


KEYWORDS 

Online discussion, triangulation, data analysis, learning. 


1. INTRODUCTION 

There has long been interest in developing forms of online collaborative learning in both formal and informal 
contexts. These developments have provided researchers with the challenge of describing and evaluating the 
learners’ experience of participation and the online archives that they create. In addressing this challenge 
researchers have developed a range of methodologies and methods, many of which can be divided between 
focus on message /focus on participant. 

Message focused analysis includes content analysis (e.g. De Wever et al. 2006); conversation analysis 
(e.g. Stahl, 2005) and discourse analysis (e.g. Littleton &Whitelock 2005; McConnell 1994). In addition, 
archives have frequently been analysed and described in respect to number of messages and breakdown of 
messages by sender and by group (e.g. gender or cohort in formal learning). There have been attempts to 
provide social network analyses too (e.g. de Laat et al. 2007; Rabbany et al. 2013) and more recently learning 
analytics (e.g. Agudo-Peregrina et al. 2014). In contrast to message analysis, person focused analysis has 
typically included interviews and surveys of learners’ attitudes to online participation, their backgrounds and 
their evaluation of their experiences online. At times, more ethnographic approaches have been undertaken, 
most notably Lindtner et al. (2008). 

Both approaches, and the particular methods within each, have their own advantages and disadvantages. 
For example, it is an obvious step to provide data on numbers taking part in online debates and the frequency 
with which individuals or groups post as these will say something about the intensity of the discussion. 
However, it is not straightforward to explore the relationship between participation and learning and a 
particular challenge that has dominated research has been content analysis of messages. Clearly the automatic 
archiving of messages has given almost unique opportunities for researchers to explore interactive learning, 
but making sense of these archives is open to different types of interpretation (De Wever et al., 2006). 
In-depth content analysis was introduced by Henri (1992) and taken forward by, amongst others, 
Gunawardena et al. (1998) who developed a model to judge the quality of online interaction and of the 
learning experience. Yet, while researchers have claimed an objectivity in their analyses doubts remain. For 
example, Naidu and Jarvela (2006, p. 101) note that “keeping the complex characteristic of human learning 
in mind, it is never possible to find full evidence of learning from ‘traces’, such as computer notes of 
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discussion threads”. Hammond (2015, p. 229) also questioned the assumptions made about participation 
arguing it was “easy to be sanguine about the affective and motivational gains from participation in these 
contexts and to identify a process of ‘knowledge building’ without asking difficult questions as to the status 
of that knowledge.” 

A further problem in over focusing on message analysis is that this may lead to the erroneous assumption 
that those who did not send messages gained nothing from reading / reflecting on others’ messages. ‘Quiet 
participation’ (or so called ‘lurking’) may be important to the maintenance of community and may be not just 
tolerated but welcomed by some active participants - something that would not be uncovered without directly 
interviewing members of forums (e.g. Takahashi et al. 2003). In contrast, by themselves surveys and 
interviews may offer rather misleading findings on participation. For example, they typically show a great 
deal of generalised support for the idea of collaborative learning which may not borne out by rates of 
participation in particular forums. 

Of course the argument for a mixed methods approach in social research has long been made (e.g. 
Johnson & Onwuegbuzie, 2004) and Dennen (2008) and Naidu and Jarvela (2006) amongst others have noted 
that those studying online learning need not stick to one method of analysis. Mixed methods enhances the 
trustworthiness of research findings by providing confirming, complementary and contrasting sources of 
data. For instance. Wee and Looi (2009) provided an example of the social construction of mathematical 
knowledge that included comparison of the researchers’ analysis to the participants’ own interpretations, de 
Laat et al. (2007) explained how they used content analysis, interviews and social network analysis to 
investigate a networked learning community, noting the value of both data and methodological triangulation. 
Schrire (2006) incorporated content analysis into a case study methodology as it helped address ‘what’ and 
‘how’ research questions. Hammond and Wiriyapinit (2005) carried out an interpretive case study using a 
variety of methods including questionnaire survey, text analysis and interviews. However, though there is, at 
least on intuitive grounds, much to recommend it, triangulation is not a routine strategy and there have only 
been sporadic attempts to reflect on its methodological possibilities. This paper then addresses a gap by 
looking at the use of a mixed methods approach to describing ‘what was going on’ in one online community. 


2. THE STUDY 

This research involves a social educational online network, namely IGGY. IGGY was created in the UK by 
the University of Warwick for academically gifted young people, aged 13 to 18. According to IGGY’s 
database, the network currently has around 7000 active members. IGGY has members from all over the 
world, though most live in the UK (n = 6547). Around 60% are 16 to 18 years old and the rest 13 to 15 years 
old. Four tenths of the total number of active members are female, 15% are male (45% did not provide this 
information). An important feature of IGGY is the high level of participation safety - for example the 
network is closed to non-members and non-disclosure of personal information is ensured through regular 
monitoring of communication by organisers. IGGY can be regarded as an unusual or unique online network 
offering a hybrid of social and individual learning. It feels open in that members tend not to know each other 
in person, but closed as students usually need to be recommended by a teacher in order to join the network 
(for a more detailed explanation, see Charalampidi et al., 2014). 

The IGGY network consists of five sections; Profile, Members, Debate, News and Events, Knowledge. Of 
particular importance to the members are the Debate and the Knowledge sections. 

The Debate section is broad and may include anything that might be of interest to the members. Debates 
can be initiated by members or mentors (these are local university students or members of the IGGY staff). 
Debates are moderated and. reflecting the ethos of IGGY, while they tend to be conversational they are also 
discursive and are seen by members as different from the everyday social networking sites in which they 
participate. Meanwhile, the Knowledge section contains learning material grouped around academic 
categories such as Maths, Science, History and Politics, and Creative Writing. IGGY does not offer its 
members a guided programme, rather members are expected to identify for themselves relevant challenges. 
These cover topics of interest to the community but are not matched against any particular awarding body’s 
programme of study. Participation in challenges is not formally assessed but is led by members of the IGGY 
team, the mentors or invited academics. 
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Researching IGGY may throw light on online participation and interaction patterns alongside the 
potential educational and/or affective benefits from participation. It also throws interesting light on the notion 
of giftedness. Underlying the various questions we posed while researching IGGY laid a wider question of 
how we could describe what was going on online. To address this question we decided to employ a mix of 
methods, including interviewing, questionnaire survey and content analysis, on an expectation that our 
understanding of IGGY would be strengthened by the unique contribution of each method. 

The approach was an iterative one. For example, in the early stage of the research, questionnaires were 
sent via email to a few members, who were then interviewed. More interviews followed which yielded 
significant findings in relation to the experience of participation (see Charalampidi et al., 2014). However, 
more data were needed and a revised questionnaire was prepared and uploaded on the network for a period of 
approximately eight months. Throughout this period a content analysis of messages from discussion forums 
was undertaken. There is not the space to present all the findings to date from our exploration of this 
network, instead this particular paper focuses on the methodology. It considers: the methods used; examples 
of using the methods; the benefits of a triangulated approach. 


3. THE METHODS USED TO ANALYSE PARTICIPATION 

In line with our earlier categorisation we look here at message focused and person focused analysis. 

3.1 Message Focused Analysis 

Analysis was carried out on posts found in the debate section of the network. IGGY had designated 16 broad 
topics for debate at the time of our analysis: Writing wrongs essay competition; Unitracks; University offer 
holders; Homework help; IGGY community hub; Help and feedback; Student mentors; Careers and personal 
development; What ’s it like to be gifted; Education and the internet; Science; Maths; History; English and 
creative writing; Politics; Law. These we grouped into four categories: cognitive; social / moral / political; 
personal development; administrative. 

The most popular of these debates were identified through analysis of numerical data including the 
number of posts and views. Some of these debates required short, quick answers such as Three Word Story?, 
First Thoughts in Mind but others were discursive covering questions such as Who Believes in Evolution and 
Why/Why not?. We decided to apply a more fine grained analysis to some of these debates including Is 
Homework A Waste Of Time?, What Is The Best Place You’ve Ever Been To On Holiday?, How Do You Tell 
If Someone Is Gifted? and Studying Law At University. These debates were representative of the cognitive, 
social / moral / political, personal development categories mentioned above, but not the administrative 
category. A further criterion for selecting debates was that they evidenced the participation of members who 
had been interviewed by the researchers. This meant that in interviews we could refer back to examples of 
debates and of participation. 

After considerable trial and error, our content analysis focused on analysing large units of meaning. The 
coding scheme was finalised after several false starts, and contained the key codes Triggering a discussion 
(T), Inviting a response (R) and Stating (S), and several sub codes (see Tablel). We wanted a scheme which 
would not be overcomplex and thus we limited our focus to just three main codes. Within our scheme we 
wanted to identify situations in which interaction was invited (the T and R codes) as interaction has been 
central to claims made about the value of online discussion (e.g. Swan, 2002). We also wanted the scheme to 
help us identify how members justified their opinion and made claims to knowledge. As classroom teachers 
we realised that we spent a great deal of our time asking learners about the moral, practical and academic 
basis for the judgements they reached and we wanted to examine how this was done online and how sources 
of knowledge were evoked in different contexts. Thus our subcategories directed us to look at how 
participants responded to particular texts and drew on external sources, general knowledge, accepted facts, 
their own experience and own value judgements to support their arguments. 
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Table 1. Codes used in content analysis 


Codes 

How achieved (sub 
codes) 

Examples 

Triggering discussion 

Introducing, 

(T/Introducing) “There are a lot 

- T 

Maintaining, Asking, 
Acknowledging 

of stereotypes surrounding 
intelligent people. How true do 
you find them?” 

(T/Maintaining) “This is really 
nice. Thanks.” 

Stating - S 

Appeal to: Reading, 
General knowledge. 

Facts, Value judgements 
( Aesthetic , Moral), Own 
experience, No reason 
given 

(S/General knowledge) 

“Driving less can have 
enormous benefits for the 
environment, while walking 
and bicycling can also improve 
your health.” (S/Value 
judgement) “I think 
academically gifted is showing 
ability in many academic 
subjects; talented is in one.” 

Responding - R 

Disagreeing, Agreeing, 
Resolving, Expanding on 
previous comments plus 
Appeal to: Reading, 
General knowledge. 

Facts, Value judgements 
(. Aesthetic , Moral), Own 
experience, No reason 
given 

(R/Disagreeing by appeal to 
own experience) "But in my 
school we usually spend so 
much time checking everyone 
has handed the h/w, we might 
as well have done the work it 
that time!!” (R/ Agreeing but no 
reason given) “I agree 
nebiyah!” 


Apart from analysing the debate transcripts in terms of functions of posts, we identified who interacted 
with whom. 

3.1.1 Examples of Message Focused Analysis 

Message focused analysis began by reading the forums and getting a feel for them. This was beneficial in 
three ways: it provided access to tangible examples of knowledge claims made by earlier interviews; it 
enabled the identification of debates that were of particular relevance to our study, and it stimulated the 
formation of interview questions that examined various aspects of these debates in more detail. 

Selected examples of debates analysed thoroughly are now presented. The first debate invited members to 
share their opinion regarding the best holiday destination they had ever been to. The second concerned the 
significance of homework and the third encouraged members to put forward any questions they might have 
had regarding studying law at University. 

Table 2 summarises the number of units of analysis labelled as interactive or non-interactive. These raw 
totals informed us about participation practices in respect to different debates. Each debate had particular 
characteristics: the first triggered a sharing of personal experiences, the second triggered particularly strong 
interaction and the third generated many information requests. This suggested that different topics provoked 
different forms of cognitive engagement. 


Table 2. Number of functions per debate 


Debate 

Functions 


What is the best place you’ve ever 

Interactive 

T = 4 

Non-interactive 

S =23 

been to on holiday? 

Is homework a waste of time? 

R= 6 

Total = 10 

T = 38 

Total = 23 

S = 105 

Studying law at University 

R= 199 

Total = 237 

T = 8 

Total = 105 

S = 1 


R = 2 

Total = 10 

Total = 1 
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Visualisation diagrams (Figures 1, 2 and 3) enabled us to identify the pattern of interactions within the 
discussions and the key participants around which discussions evolved. In the figures, the square nodes 
represent learner members of IGGY, the circles represent mentors or members of the IGGY staff, and the 
lines represent connections between the nodes. The size of the nodes is proportional to the number of their 
connections. The octagon signifies those messages that did not address a particular discussant but rather all 
discussants in the debate. Using these diagrams we could explore whether online participation could be better 
described as ‘many to many’ interaction or one to many or as simply chaotic. 

It was interesting to observe that even though all messages revolved around the initial post, the participant 
who triggered the discussion in all three debates did not contribute further to it. It was also clear that the 
mentors in the first and second debate (see Figure 4) were particularly active in sending messages and were 
frequently addressed when members replied. This suggested that the mentors’ contribution in encouraging 
further interaction among discussants was significant. The second debate is of particular interest as it has 
been one of the most popular in IGGY. In this debate, 122 students and 9 mentors participated. Figure 3 
shows that apart from the main discussion, several subgroup discussions were developed. Many messages 
were also directed to the group as a whole. This suggested that discussants in this debate were not only 
interested in the topic as such, but also in the opinions expressed by others. 



Figure 1. ‘What is the best place you’ve ever Figure 2. Is homework a waste of time - representation of interactions 
been to on holiday?’ - representation of 
interactions and functions of posts 



Figure 3. Studying law at University - Figure 4. Is homework a waste of time - representation of M2's 

representation of interactions and functions connections 

of posts 
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3.2 Person Focused Analysis 

A survey (n = 76 responses) was carried out comprising of 25 questions; 22 closed questions, 2 open-ended 
questions and 1 question that invited students to opt in for an interview. The closed questions included Likert 
scales, yes - no questions and multiple choice questions. The questions were broadly divided in two 
categories; questions about the students’ profile (e.g. gender, age etc.), and his/her online experience. The 
latter covered the themes of online behaviour and forms of engagement and provided quantitative data on 
issues such as membership duration, the frequency of accessing the network, the time spent using the 
network during a typical week, and the frequency of engagement with various types of activities. It also 
examined the members’ preference over interactive or non-interactive activities, perceived benefits and 
reasons for using IGGY, feelings towards the community, constraints and suggestions for improving the 
online experience. 

The use of the network was further explored through a series of semi structured interviews (n = 12). Key 
themes that emerged concerned the users’ profiles (hobbies, family, friends), the idea of giftedness 
(conceptions of giftedness, the label, feelings and/or problems related to it), their use of technology in 
general, their use of IGGY (expectations, why join, why use, what do you do, benefits, online relationships 
and community, facilitators, constraints, suggestions for improvement). In the latest round of interviews we 
also used one strategy, stimulated recall, which enabled us to discuss intentions and composition with some 
interviewees in relation to particular debates. 

The questionnaire survey presented us with an overview of the use of IGGY. From the survey we found 
that members carried out a variety of activities in IGGY but the favourite ones were reading and/or replying 
to debates and doing quizzes. Members used IGGY for a number of reasons: to address lack of challenge at 
school, to access learning resources, to meet new people, to communicate with other members, and to learn 
about other cultures. Many members stated that they experienced both educational (i.e. vocabulary 
development), cultural (i.e. knowledge of other cultures) and affective benefits (i.e. confidence in expressing 
their opinion) through their participation. In general, IGGY was seen as an educational community within 
which members felt trust, empathy and respect. The main constraint in using IGGY was lack of time and 
learning to navigate the network. 

Interviews provided the detail for this general picture. For example, one interviewee (coded in our study 
as IGGYFemalel3) was classified as a frequent user of the network as she accessed it daily, spending 
between one and two hours in it. IGGYFemalel3 provided explanations and examples to support her idea 
that the network was helpful and valuable for her. She also expanded on debates to which she had 
contributed, read or initiated. She participated when she found the topic important and challenging, as in the 
second debate above. She was led to contribute further when different views were expressed: 

“ Yeah that homework is really important for our learning to progress. I just thought that I tried to make 
people see different views. People who thought that it wasn ’t important I tried to make them see that it 
actually is really important.” 

Notably, she replied to posts when she felt she had something to add to the conversation. Yet, even when 
she remained quiet, she did read and contemplate the messages giving us insight into the process of quiet 
participation: 

“I just wanted to push it as far as possible so that they thought of different ways, but the others were so 
thought through I didn ’t know what to answer. There was nothing I could say, because it was just so well 
written and so well researched. I did look at them, I did come back to it. ” 

“. . . on the most serious debates I do tend to read every single one to see just the different opinions. I 
don ’t always post in them but I read other peoples ’ experiences .” 

The interview allowed this IGGY member to reflect on the size of debates (small versus large scale) and 
type (i.e. fun versus serious). She felt that a debate that generated carefully considered replies was successful 
even if the number of replies was limited. Additionally, she valued any type of debate, being fun or more 
serious, if they had something to offer her: 

“. . . there weren ’t that many replies but the replies that were there were really thorough and thought 
through so that is whcit I was trying to get people to do, to think about it and give me an honest answer. ” 

“/ like the ones that really challenge my way of thinking, people who try to convince me that their way of 
thinking is better because I can argue with them. They sometimes even convince me! I also like the ones that 
are fun because it’s just really nice to take some time out of serious things and just have fun even though it 
still brings something to me.” 
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She also shed further light on facilitators of participation and referred to the importance of social 
presence. She believed that replying to specific members was useful in making them feel both accepted and 
confident as “if shows that somebody has actually taken the time to read their message”. She explained that 
she really enjoyed the year and a bit now (she) spent on IGGY and (she) want(ed) as many members to 
feel welcome to the community as (she) was.” 

The interview confirmed the earlier finding regarding the importance of the mentors’ participation. 
IGGYFemalel3 commented on this role and stated her appreciation of their contributions. She even referred 
to two specific mentors, one of whom was the mentor (M2) who stood out in the second debate above: 

“/ think they bring a high level of sophistication and a lot of intelligence to IGGY and their posts are 
really interesting. One of the usernames (...) I think that’s her name, always wrote specifically to someone 
for example she put a username and answered, and there could be 5 messages for different members on one 
debate and I think that was really good that she took the time to answer.” 

Finally, via the interview we were able to reach an understanding of what online learning meant to the 
members. IGGYFemalel3 felt that online learning included the exchange of different views and the 
stimulation of rethinking about one’s own ideas. Flence, she did perceive her participation in debates as 
learning, including debates on less academic topics: 

“I’d say it is because, from other peoples ’ experiences and other peoples ’ views it just brings on a whole 
other way of your thoughts and maybe you thought one way but somebody else thought another way and it 
just makes you think, so it is challenging your brain which is what my definition is of learning and, just some 
debates do relate a lot to learning but other debates are slightly more like fun and 1 think they are also 
important. One of the debates, I think it ’s “Five random facts about you ”, you just say the five first things 
about you that pop into your head and it’s just funny to see what people write but others like the one about 
geeks and nerds, they just really make you think, make you learn and share experiences.” 

One obvious limitation of the interviews and even the survey was that of sampling. Not surprisingly those 
that volunteered to be interviewed tended to be among the most active of members and their experiences 
might not be representative. The survey was likely to be more representative but to date take up has been less 
than we would like. 


4. CONCLUSIONS 

The paper began by noting the variety of approaches to analysing online participation. Two main approaches 
were identified; message focused and person focused analysis. In our study we combined these approaches to 
exploit the opportunities afforded by each. We drew three key conclusions from this attempt to apply a 
triangulated approach to understanding ‘what is going on online’. 

First, different sources of evidence provide different insight. In particular, the message focused analysis 
informed us about the structure of debates and showed how debates were triggered, who triggered them, who 
contributed and how. The analysis gave clues as to how discussions were sustained and pointed to the key 
role of moderators. Our content analysis gave us insight into the different sources of knowledge and claims to 
knowledge and how these differed depending on the nature of the discussion. This was important as a claim 
to academic knowledge needed to be founded on more than personal experience and should consider 
appropriate evidence. However, such analysis did not provide access to the participants’ perceptions about 
what is happening online but rather an interpretation from an ‘external’ point of view. Thus the need for 
interviews, to allow an in-depth exploration of the participants’ experiences and offer answers to ‘why’ 
questions. Interviews have the additional potential of informing researchers about ‘hidden’ or ‘quiet’ 
participation. Alongside interviews, surveys can enable access to a wider population and provide background 
information, both quantitative and qualitative. This can be beneficial in examining individual members or in 
identifying subgroups with common characteristics. 

Second, the analysis of an online environment should not be treated mechanistically. For example, coding 
for content analysis was not chosen ‘off the peg’ but rather developed by ourselves to fit around the questions 
we wanted to ask. More importantly while we used familiar methods of contrast, consistency and 
complementarity to triangulate findings this required a continual cross checking of different data rather than a 
simple aggregation. Indeed, based on constant comparison of data we were able to reach the conclusion that 
IGGY can be described as an educational community in which, through participation and interaction, 
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members experience learning benefits, albeit with constraints on members’ participation and differentiated 
patterns of participation. 

Third, a triangulated approach is intensely time consuming and perhaps this explains its uneven use in the 
field. Yet the approach is a valuable one and we are in danger of making misleading claims about online 
learning if we rely on only one source of data. 
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